PHYSTAT2003, SLAG, Stanford, California, September 8-11, 2003 



1 



Blind Analysis in Particle Physics 

Aaron Roodman 

Stanford Linear Accelerator Center, Stanford, CA 94025, USA 



A review of the blind analysis technique, as used in particle physics measurements, is presented. The history of 
blind analyses in physics is briefly discussed. Next the dangers of experimenter's bias and the advantages of a 
blind analysis arc described. Three distinct kinds of blind analysis in particle physics are presented in detail. 
Finally, the BABAR. collaboration's experience with the blind analysis technique is discussed. 



X 



1. Introduction 

A blind analysis is a measurement which is per- 
formed without looking at the answer. Blind analyses 
are the optimal way to reduce or eliminate experi- 
menter's bias, the unintended biasing of a result in a 
particular direction. 

In bio-medical research the double-blind random- 
ized clinical trial is the standard way to avoid bias. 
In such experiments both patients and clinicians are 
blind to the individual assignments of treatments 
to patients, and that assignment is made randomly. 
A double-blind randomized trial was first used in 
1948 by Hill in a study of antibiotic treatments for 
tuberculosis 0. Amazingly, the concept of a double- 
blind trial dates back to at least 1662, when John Bap- 
tista van Helmont made the following challenge lj: 

Let us take out of the hospitals,... 200, 
or 500 poor People, that have Fevers, 
Pleurisies, etc. Let us divide them into 
half, let us cast lots, that one half of them 
may fall to my share, and the other to 
yours; I will cure them without blood- 
letting and sensible evacuation... We shall 
see how many funerals both of us shall 
have. But let the reward of the contention 
or wager, be 300 florens, deposited on both 
sides... 

A notable early use of a blind analysis in physics 
was in a measurement of the e/m of the electron, by 
Dunnington 0. In this measurement, the e/m was 
proportional to the angle between the electron source 
and the detector. Dunnington asked his machinist to 
arbitrarily choose an angle around 340°. Only when 
the analysis was complete, and Dunnington was ready 
to publish a result, did he accurately measure the hid- 
den angle. 



2. Experimenter's Bias 

Experimenter's bias is defined as the unintended in- 
fluence on a measurement towards prior results or the- 
oretical expectations. Next, we consider some of the 
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Figure 1: This cartoon illustrates how a result may vary, 
statistically, for different arbitrary choices of a cut. 



ways in which an unintended bias could be present in 
a measurement. 

One scenario involves the choice of experimental se- 
lection requirements, or cuts. Often, a measurement 
may be equally well done, in terms of sensitivity or 
uncertainties, with a range of values for a particular 
selection cut, and the exact cut value used may be 
chosen arbitrarily. This is illustrated in the cartoon 
in Fig ^ where there is a plateau in the sensitivity, 
and the cut value could be chosen anywhere in that 
plateau. However, the value of the result may vary, 
typically within the statistical uncertainty, depending 
on the exact value of cut chosen. If the cuts are set 
with knowledge of how that choice affects the results, 
experimenter's bias could occur. In this case, the size 
of the bias could be on the order of the statistical 
uncertainty. 

Another, less subtle, scenario involves measure- 
ments of small signals, such as the search for rare 
processes or decays. Here experimenter's bias could 
occur if the event selection is determined with prior 
knowledge of the effect of that selection on the data. 
One danger is that the selection cuts can be tuned 
to remove a few extra background-like events, yield- 
ing a result biased to lower limits. Another danger is 
that the cuts can be tuned to improve the statistical 
significance of a small signal. 

In general, experimenter's bias may occur if obtain- 
ing the correct result is the standard used to evalu- 
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Figure 3: Hidden signal box from a search for the decay 
K° L -> from Ref. ||. 



Figure 2: Summary of B meson lifetime ratio 
measurements. The average has a \ 2 =4.5 for 13 degrees 
of freedom. 



reduce or eliminate this bias is needed. 



ate the quality of the measurement. The validity of 
a measurement may be checked in a number of ways, 
such as internal consistency, stability under variations 
of cuts, data samples or procedures, and comparisons 
between data and simulation. The numerical result, 
and how well it agrees with prior measurements or the 
Standard Model, contains no real information about 
the internal correctness of the measurement. If such 
agreement is used to justify the completion of the mea- 
surement, then possible remaining problems could go 
unnoticed, and an experimenter's bias occur. 

Does experimenter's bias occur in particle physics 
measurements? Consider the results on the ratio B 
meson lifetimes shown in Figure |21 The average has a 
X 2 — 4.5 for 13 degrees of freedom; a % 2 this small or 
smaller occurs only 1.5% of the time. At this level, the 
good agreement between measurements is suspicious, 
but for each individual result no negative conclusion 
should be made. Nonetheless, it can be argued that 
even the possibility of a bias represents a problem. 
The PDG[3| has compiled a number of measurements 
that have curious time-histories. Likewise, while it is 
difficult to draw negative conclusions about a single 
measurement, the overall impression is that experi- 
menter's bias does occur. Finally, there are numer- 
ous examples in particle physics of small signals, on 
the edge of statistical significance, that turned out to 
be artifacts. Here too, experimenter's bias may have 
been present. 

In all of these cases, the possibility of experimenter's 
bias is akin to a systematic error. Unlike more typi- 
cal systematic effects, an experimenter's bias cannot 
be numerically estimated. Therefore, a technique to 



3. Blind Analysis 

A Blind Analysis is a measurement performed with- 
out looking at the answer, and is the optimal way to 
avoid experimenter's bias. A number of different blind 
analysis techniques have been used in particle physics 
in recent years. Here, several of these techniques are 
reviewed. In each case, the type of blind analysis is 
well matched to the measurement. 

3.1. Hidden Signal Box 

The hidden signal box technique explicitly hides the 
signal region until the analysis is completed. This 
method is well suited to searches for rare processes, 
when the signal region is known in advance. Any 
events in the signal region, often in two variables, are 
kept hidden until the analysis method, selection cuts, 
and background estimates are fixed. Only when the 
analysis is essentially complete is the box opened, and 
an upper limit or observation made. 

The hidden signal box technique was used 1 in a 
search for the rare decay K® — ► // ± e T . This decay 
was not expected to occur in the Standard Model, 
and the single event sensitivity of the experiment was 
one event in 10 11 K® decays. Any signal was expected 
inside the box in M^ e and shown in Figure [3J the 
possible contents of this box were kept hidden until 
the analysis was completed 0. 



lr niis is the first use known to the author. 
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The use of this method is now a standard method 
for rare decay searches, when the signal region is 
known in advance. One additional subtlety lies in the 
size of the hidden box. Generally, the box is initially 
chosen to be somewhat larger than the signal region, 
so that the final signal cuts may be chosen without 
bias as well. Otherwise, this technique is straightfor- 
ward to apply. 

3.2. Hidden Answer 

For precision measurements of parameters, a dif- 
ferent technique for avoiding bias must be used. In 
this case, hiding the answer is often the appropriate 
method. The KTeV experiment used this technique 
in its measurement of e'/e. The value of e'/e was 
found in a fit to the data, and a small value of or- 
der 10 — 10~ 3 was expected. In this case, KTeV 
inserted an unknown offset into its fitting program, so 
that the result of the fit was the hidden value: 

e'/e (Hidden) = j ^ J x e'/e + C (1) 

where C was a hidden random constant, and the 
choice of 1 or —1 was also hidden and random. The 
value of the hidden constant, C, was made by a 
pseudo-random number generator with a reasonable 
distribution and mean. KTeV could determine its 
data samples, analysis cuts, Monte-Carlo corrections, 
and fitting technique while the result remained hid- 
den, by looking only at e'/e (Hidden). The use of the 
1 or — 1 factor prevented KTeV from knowing which 
direction the result moved as changes were made. In 
practice, the result |5| was unblinded only one week 
before the value was announced. 

The hidden answer technique is well-suited to pre- 
cise measurements of a single quantity. The complete 
analysis, as well as the error analysis, may proceed 
while blind to the result. An additional consideration 
is whether there are any distributions which will give 
away the blinded result. Often the exact value of the 
measurement is not readily apparent from the relevant 
plots; in this case those plots can be used without is- 
sue. 

3.3. Hidden Answer and Asymmetry 

For certain measurements hiding the answer is not 
sufficient; it may also be necessary to hide the visual 
aspect of the measurement. One example is an asym- 
metry measurement, such as the recent CP-violation 
measurement by BABAR. In this case, the rough size 
and sign of the asymmetry can be seen by looking at 
the At distributions for B° and B° decays into CP 
eigenstates, as shown in Figure^. Before CP viola- 
tion had been established, and to avoid any chance of 
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Figure 4: The At distributions for B decays into CP 
eigenstates, for sin2/? = 0.75 with the B° flavor tagging 
and vertex resolution which are typical for the BABAR 
experiment, a) The number of B° (solid line) and B° 
(dashed line) decays into CP eigenstates as a function of 
At. b) The At min d distributions for B° (solid) and B° 
(dashed). 

bias, a blind analysis was developed to hide both the 
answer and the visual asymmetry. 

In BABAR's CP- violation measurement the result, 
found from a fit to the data, was hidden as in Equa- 
tion In addition, the asymmetry itself was hidden 
by altering the At distribution used to display the 
data.0 To hide the asymmetry the variable: 



At (Blind) = 




x s Tag x At + Offset (2) 



was used to display the data. The variable STag is 
equal to 1 or —1 for B° or B° flavor tags. Since 
the asymmetry is nearly equal and opposite for the 
different B flavors, we hid the asymmetry by flipping 
one of the distributions. In addition, the CP- violation 
can be visualized by the asymmetry of the individual 
B° and B° distributions. In turn, this was hidden by 
adding the hidden offset which has the effect of hiding 
the At = point. The result is shown in Figure QJa, 
where the amount of CP- violation is no longer visible 
(the remaining difference is due to charm lifetime ef- 
fects). Also it is worth noting that for a given data 
sample, due to statistical fluctuations, the maximum 
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of the distribution will not exactly correspond to the 
At = point, as in the smooth curves shown. 

This blind analysis technique allowed BABAR to use 
the A^Biind distribution to validate the analysis and 
explore possible problems, while remaining blind to 
the presence of any asymmetry. There was one ad- 
ditional restriction, that the result of the fit could 
not be superimposed on the data, since the smooth 
fit curve would effectively show the asymmetry. In- 
stead to assess the agreement of the fit curve and the 
data, a distribution of just the residuals was used. In 
practice, this added only a small complication to the 
measurement. However, after the second iteration of 
the measurement, it became clear that the asymme- 
try would also remain blind if the only At distribution 
used was of the sum of B° and B° events, and that 
no additional checks were needed using the individual 
At distributions. 

3.4. Other Blind Methods 

The kinds of measurements already discussed, such 
as rare searches and precision measurements of phys- 
ical parameters, are well suited to the blind analy- 
sis technique. Other kinds of analyses are difficult to 
adapt to the methods described. For instance, branch- 
ing fraction measurements typically require the careful 
study of the signal sample in both data and simula- 
tion, so it is not possible to avoid knowing the number 
of signal events or the efficiency. In this case, other 
techniques may be considered. One method is to fix 
the analysis on a sub-sample of the data, and then 
used the identical method on the full data sample. 
One may argue about the correct amount of data to 
use in the first stage, too little and backgrounds or 
other complications may not be visible, too much and 
the technique loses its motivating purpose. Another 
method is to mix an unknown amount of simulated 
data into the data sample, removing it only when the 
analysis is complete. 

Another difficult example is the search for new par- 
ticles, or bump-hunting. In this case, since the signal 
region is not known a-priori, there is no one place 
to put a hidden signal box. However, such measure- 
ments may be the most vulnerable to the effects of 
experimenter's bias. Certainly, there is some history 
of statistically significant bumps that are later found 
to be artifacts. The possibility of using a blind anal- 
ysis technique may depend on the understanding of 



the relevant background. If the background can be 
estimated independently of the bump-hunting region, 
than the analysis and selection cuts may be set in- 
dependently of the search for bumps. Here again is 
a case in which the exact method used must be well 
matched to the measurement in question. 

4. Conclusion 

The experience of the BABAR collaboration in us- 
ing blind analyses is instructive. While the collabora- 
tion had initial reservations about the blind analysis 
technique, it has now become a standard method for 
BABAR . Often the blind analysis is a part of the in- 
ternal review of BABAR results. Results are presented 
and reviewed, before they are unblindcd, and changes 
are made while the analysis is still blind. Then when 
either a wider analysis group or an internal review 
committee is satisfied with the measurement the re- 
sult is unblinded, ultimately to be published. With 
several years of data taking, and many results, BABAR 
has successfully used blind analyses. 
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